Multi-Shelf System Clock Synchronization 
Field of the invention 

[01] The invention relates to synchronizing content transport in 
communications networks, and in particular to methods and apparatus for 
employing any master clock signal received at a multi-shelf system to 
synchronize content transport via the multi-shelf system. 

Background of the invention 

[02] In the field of communications, ongoing research and development 
concerns master clock signal distribution and content transport 
synchronization. 

[03] In communications networks, a signal transport delay is incurred when 
content is conveyed over communications links between communications 
network nodes. Particularly as content is being conveyed at high bit rates, the 
ability of communications network nodes to synchronize content transport is 
diminished: on one hand, due to very short bit transmission time periods at 
high transport rates, and on the other hand, due to master clock signal source 
instabilities. 

[04] As content transport over communications networks encompassing the 
Earth incur signal transport delays in the order of seconds, synchronization to 
an absolute clock signal is hard to achieve and ensure. Attempts have been 
made towards this end including proposing the use of Global Positioning 
System (GPS) signals as a master clock signal reference. However, there are no 
assurances that GPS signals can be received without fail at all times under all 
conditions. 
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[05] In the absence of absolute clock signal synchronization, signal 
synchronization becomes very important. The absence of signal 
synchronization in conveying content leads to an unbound signal skew 
introducing further content transport delays and overheads. The absence of 
signal synchronization in conveying content in the communications 
infrastructure also leads to an uncontrollable signal transmission and reception 
jitter which may put detrimental short high demands on storage at intermediary 
nodes in transport paths. The exemplary uncontrollable jitter and the unbound 
signal skew mentioned herein above are but a few of the effects typically 
experienced. 

[06] Towards achieving synchronization, clock standards such as Cesium 
clocks are typically employed to provide master clock signals. Cesium clock 
generated master clock signals are distributed to individual network nodes in 
communications networks via designated interconnecting links. While 
redundancy is a requirement, Cesium clock sources are expensive and if too 
many are used in a communications network, the synchronization of the 
multiple Cesium clocks becomes the stumbling block. As network nodes are 
designed to be multiply and redundantly interconnected in communications 
networks, it would be impossible to ensure that the content transport paths 
follow master clock signal distribution paths. Nor can it be ensured that master 
clock signal distribution paths follow content transport paths. Therefore, 
inevitably, active synchronization must be performed at each network node in a 
communications network. 

[07] With multiple master clocks in a communications network, multiple 
master clock signals are typically received at each network node. Therefore 
active signal synchronization at each network node must include selecting 
which master clock signal received to be used to phase align content 
conveyance thereto. Being able to select between multiple sources is desired as 
master clock signal sources may fail, become unreliable, or at times need 
servicing; and/ or as the clock signal distribution infrastructure may experience 
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failure. In accordance with various interconnectivity standards, at least two 
master clock signals must be provided to each network node in a 
communications network: one being designated as the primary master clock 
signal to which content transport is actively synchronized to, and the other 
designated as the secondary standby master clock signal to which content 
transport is to be synchronized to should the primary master clock signal be 
unavailable or experience degradation. 

[08] Take for example a typical single-shelf network node implementation, 
the single-shelf network node typically has a backplane design which includes a 
switching fabric with interface card connectors for interfacing with line cards 
having physical ports thereon. Communications links connect directly to the 
physical ports. An exemplary Bellcore Stratum-3 compliant 8kHz System 
SYNChronization signal (SSYNC), is generated by a System Synchronization 
Unit (SSU) at each network node. The generated Bellcore Stratum-3 compliant 
8kHz SSYNC signal is frequency locked and phase locked to a selected, 
externally generated, master clock signal (External SYNChronization signal 
(ESYNQ) received via one of the physical ports of the single shelf network 
node. The frequency-locked phase-locked SSYNC signal is subsequently 
distributed to each line card associated with the single shelf network node for 
use in conveying content. For the purpose of ESYNC signal collection and 
SSYNC signal distribution, the backplane design includes special interface card 
special traces thereon and connector pins, to collect and distribute analog clock 
signals within the single shelf network node. 

[09] Recent developments in the art include: 

[10] Prior art United States Patent 5,910,753 entitled "Direct Digital Phase 
Synthesis" which issued on June 8 th , 1999 to Bogdan describes the controlled 
generation of clock signals for telecommunications applications in a single shelf 
network node from a control voltage signal. 
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[11] In a prior art United States Patent Application serial number 10/087,521 
entitled "Direct Digital Synthesizer Phase Locked Loop" published on 
December 5 th , 2002, O'Leary et al. describe an exemplary direct digital phase 
locked loop implementation (DPLL) having a numerically controlled oscillator 
employed in clock generation in a shelf controller of a single shelf network node 
system implementation. The DPLL synchronizes the system clock to a selected 
reference clock signal from a shelf element such as an interface card. 

[12] A co-assigned prior art United States Patent 5,638,410 entitled "Method 
and System for Aligning the Phase of High Speed Clocks in 
Telecommunications Systems" which issued on June 10 th , 1997 to Kuddes 
describes the use of a digital delay line to phase align standby and primary 
clocks to minimize phase-related effects of a clock switchover in a single shelf 
telecommunications system should the primary clock experience a failure. 

[13] The continued increasing demand for higher content transport 
bandwidth has led to multi-shelf network node designs. As a side effect, the 
issues related to content transport synchronization and clock signal distribution 
in communications networks have re-emerged in respect of multi-shelf network 
node design and implementation. Known solutions employed in single-shelf 
network node systems cannot be employed in respect of multi-shelf network 
node systems largely because multi-shelf network node designs are optimized 
for content transport: 

[14] Making reference to FIG. 1, a multi-shelf network node system 100 
includes a control shelf 102, a switching shelf 104, and multiple peripheral 
shelves 106. The peripheral shelves 106 primarily exchange content with the 
communication network(s) in which the multi-shelf network node 100 
participates. The control shelf 102 can be employed exclusively for content 
transport control and may also participate in content transport. 

[15] From the point of view of content switching, the shelves of the multi- 
shelf system 100 have a star interconnection architecture with the switching 
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shelf 104 as the root node. The switching shelf 104 includes a master switching 
fabric which switches all traffic between the peripheral shelves 106 (including 
the control shelf 102 if the control shelf 102 has line cards 118 installed therein). 
Each peripheral shelf 106 has a Fabric Card (FC) 108 to which all inter-shelf 
traffic is multiplexed and from which all inter-shelf traffic is demultiplexed. A 
high bandwidth data link 110, typically an optical fiber trunk, conveys content 
between at least one fabric card 108 at a peripheral shelf 106(/102) and at least 
one Switch Access Card (SAC) 114 at the switching shelf 104. Typically little 
else is provisioned between the peripheral shelves 106(/102) and the switching 
shelf 104 by design, as data transport optimizations are sought. 

[16] From the point of view of control, the shelves of the multi-shelf system 
100 have a star interconnection architecture with the control shelf 102 as the root 
node. The control shelf 102 includes a backplane and control cards 112 
providing control layer services to the entire multi-shelf network node system 
100. Control Services Links (CSL) 116 convey control information between the 
control card 112 at the control shelf 102, and a corresponding shelf controller 
card 115 at each peripheral shelf 106. 

[17] It is understood that the entire multi-shelf system interconnectivity 
infrastructure is redundant. While redundant shelf controller, control, and 
fabric cards are shown in FIG. 1, redundant links have been omitted from FIG. 1 
to simplify the diagram. 

[18] In order to achieve high levels of throughput at the multi-shelf network 
node system 100, the peripheral shelves 106(/102) are optimized for content 
switching and access to the switching shelf 104. Therefore, control services links 
116 typically have low bandwidths when compared with the data links 110. 
Whereas the data links 110 typically include plain optical links, the control 
services links 116 may provide more functionality particularly in a multi- 
protocol multi-shelf network node deployment. 
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[19] For example, due to the current market-driven necessity to consolidate 
voice and data communications, each control services link 116 may include an 
El/Tl trunk. El/Tl trunks are typically used for Time Division Multiplexed 
(TDM) transmission of digitized voice samples and voice communications 
control information in telephone networks. El/Tl-based transmissions adhere 
to a TDM frame format having frame markers recurring at an 8kHz rate such 
that voice sample data corresponding to a particular service-level connection is 
found in the same bit position with respect to the frame marker. The 
standardized intended use of the Bellcore Stratum-3 compliant 8kHz SSYNC 
signal relates directly to El/Tl frame marker generation and detection - the 
analog characteristics of the frame marker being used in clock synchronization. 

[201 Whereas dedicated clock pins are employed by each interface card 
connector, and dedicated clock signal traces are employed on the backplane of a 
single shelf network node, a problem exists in respect of multi-shelf network 
node implementations 100 in bringing all clock signals derived from external 
communications links connected to physical ports associated with all peripheral 
shelves 106 to the SSU on the control card 112 of the control shelf 102, as a 
reference master clock signal, to synchronize content conveyance in the multi- 
shelf network node 100 thereto. For this reason, in a typical multi-shelf system 
100 only master clock signals received via interface (line) cards (118) physically 
connected to the control shelf 102 can be used for content transport 
synchronization thereto. In the field, this restriction is seen as a hindrance in 
engineering and deploying communications networks employing multi-shelf 
network nodes 100. 

[21] There therefore is a need to solve the above mentioned issues. 
Summary of the invention 

[22] In accordance with an aspect of the invention, a comparator is provided 
on a shelf controller of a peripheral shelf of a multi-shelf network node for 
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determining a phase difference information between a selected ESYNC signal 
received via a physical port associated with the peripheral shelf by a selector 
and an SSYNC signal distributed from a control shelf. Transmission means are 
employed to convey the phase difference information to the control shelf. 
Employing the selector and the comparator provides flexibility in conveying 
any ESYNC signal from any interface card associated with the any peripheral 
shelf to the control shelf of the multi-shelf network node over a reduced 
infrastructure. 

[23] In accordance with another aspect of the invention, an ESYNC phase 
difference information encoder associated with the peripheral shelf controller 
digitally encodes the phase difference information into digitally encoded phase 
error words for transmission to the control shelf via a control layer 
infrastructure. 

[24] In accordance with a further aspect of the invention, a controller 
employed in a control shelf of a multi-shelf communications network node 
includes reception means receiving a plurality of phase difference information 
streams, and a selector for selecting a phase difference information stream from 
the plurality of phase difference information streams. The controller further 
includes an ESYNC signal regenerator for regenerating an ESYNC signal from a 
SSYNC signal and digitally encoded phase difference information. The selector 
and the ESYNC signal regenerator providing flexibility in receiving at the 
control shelf any ESYNC signal from any peripheral shelf associated with the 
multi-shelf network node. Employing the selector further reduces the 
complexity of the control shelf. 

[25] In accordance with a further aspect of the invention, a method of 
conveying an ESYNC signal received via a peripheral shelf of a multi-shelf 
network node to a control shelf thereof is provided. A plurality of ESYNC 
signals are received from a corresponding plurality of interface cards associated 
with a peripheral shelf. An ESYNC signal is selected from the plurality of 
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ESYNC signals received. Phase difference information is derived from a 
comparison between the selected ESYNC signal and a System SYNC (SSYNC) 
signal provided from the control shelf. Phase difference information regarding 
the selected ESYNC signal is conveyed from the peripheral shelf to the control 
shelf. Selecting the ESYNC signal and deriving phase difference information at 
the peripheral shelf provides flexibility in conveying ESYNC signals from 
interface cards associated with the peripheral shelf to the control shelf of the 
multi-shelf network node over a reduced infrastructure. 

[26] In accordance with a further aspect of the invention, the method further 
includes digitally encoding the phase difference information and digitally 
conveying the phase difference information between the peripheral shelf and 
the control shelf. 

[27] In accordance with yet another aspect of the invention, the method 
further includes selecting a digitally encoded phase difference information 
stream from a plurality of digitally encoded phase difference information 
streams received at the control shelf, and deriving a corresponding ESYNC 
signal based on the SSYNC signal and the selected phase difference information 
stream. 

[28] Advantages are derived from a scalable synchronization signal 
infrastructure providing any selected ESYNC signal received via any interface 
card associated with any peripheral shelf of a multi-shelf network node to the 
control shelf for synchronizing an SSYNC signal thereto. 

Brief description of the drawings 

[29] The features and advantages of the invention will become more apparent 
from the following detailed description of the exemplary embodiments with 
reference to the attached diagrams wherein: 
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FIG. 1 is a schematic diagram showing network node shelf elements 
implementing a typical multi-shelf communications network node system; and 

FIG. 2 is a schematic diagram showing exemplary relationships and 
clock signal processing step in a multi-shelf communications network node 
system in accordance with an exemplary embodiment of the invention. 

[30] It will be noted that in the attached diagrams like features bear similar 
labels. 

Detailed description of the embodiments 

[31] It is desired that a multi-shelf network node design and operation enable 
the selection of a master clock signal received via any interface card (line card 
118) of either the control shelf 102 or any of the peripheral shelves 106 of a 
multi-shelf network node 100. 

[32] In view of the fact that only one El/Tl trunk is available in the control 
services link 116 which could potentially bring a master clock signal, and 
particularly the analog characteristics thereof, from a peripheral shelf 106 to the 
control shelf 102, one solution would be to provide an El/Tl trunk in a control 
services link 116 per peripheral shelf interface card connector. Such a solution 
would however suffer from: the necessity of deploying and maintaining a large 
number of El/Tl trunks, a reduced port density, a complicated shelf controller 
115 and control card 112 design, a dedicated non-standard shelf controller and 
control card connector design, a complicated control shelf backplane signal 
trace design, El/Tl trunk length dependent incurred analog signal jitter and 
signal skew, etc. necessary to bring together all received clock signals to the SSU 
for possible use in content transport synchronization. Such a solution would 
not scale with the number of peripheral shelves per multi-shelf network node. 

[33] Making reference to an exemplary implementation of an exemplary 
embodiment of the invention schematically shown in FIG. 2, an SSYNC signal 
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generated by an SSU 122 associated with a corresponding control card 112, is 
frequency locked and phase locked to an External SYNC (ESYNC) signal 
provided thereto as will be described herein below, and the SSYNC signal is 
distributed to line cards 118 of the control shelf 102, if any, directly via existing 
clock signal traces on the control shelf backplane, then via existing clock pins of 
interface card connectors; and also distributed to all peripheral shelves 106 via 
the down-link frame markers of the El/Tl trunk of each control services link 
116. 

[34] At each peripheral shelf 106, the SSYNC signal is extracted at each 
peripheral shelf 106 from down-link frame marker of the El/Tl trunk of the 
corresponding control services link 116, and distributed to each line card 118 of 
the peripheral shelf 106 via dedicated peripheral shelf backplane clock signal 
traces and interface card connector pins. 

[35] While there is a single control shelf 102 in the multi-shelf network node 
100, it is understood that the entire control infrastructure described herein 
above is redundant: there two SSUs 122 (one on each control card 112) and 
paired control services links 116 from the pair of control cards 112 at the control 
shelf 102 connect to paired shelf controller cards 115 at each peripheral shelf 106 
(and the switching shelf 104). 

[36] Externally generated master clock signals may be received via line cards 
118 associated with the control shelf 102, if any, and are provided directly as 
ESYNC signals to the SSU 122 for selection in frequency locking and phase 
locking the SSYNC signal thereto. 

[37] In accordance with the exemplary embodiment of the invention, 
externally generated master clock signals may also be received via any 
(interface) line card 118 associated with any of the multiple peripheral shelves 
106 of the multi-shelf network node 100. At each peripheral shelf 106, received 
master clock signals are provided as ESYNC signals to a ESYNC signal selector 
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119 associated with a shelf controller card 115 via dedicated clock interface card 
pins and dedicated peripheral shelf backplane clock signal traces. 

[38] In accordance with the exemplary embodiment of the invention, two best 
ESYNC clock signals are selected at each peripheral shelf 106 to be provided to 
the control shelf 102 for potential frequency lock and phase lock of the SSYNC 
signal thereto. 

[39] In accordance with the exemplary embodiment of the invention, a 
selected ESYNC signal is provided to a comparator 120 employed in 
determining a corresponding phase difference between the selected ESYNC 
signal and the SSYNC clock signal distributed by the control shelf 102. 

[40] In accordance with the exemplary embodiment of the invention, at each 
shelf controller card 115, derived phase difference information corresponding to 
a selected ESYNC signal is provided to an ESYNC phase difference encoder 124 
which digitally encodes the phase difference information. 

[41] In accordance with the exemplary embodiment of the invention, the 
digitally encoded phase difference information corresponding to each ESYNC 
signal selected at each peripheral shelf 106 is conveyed to the control shelf 102 
digitally over the corresponding control services links 116. In accordance with 
an exemplary implementation, the phase difference information is encoded in 
streams of phase error words. If the control services link 116 has multiple 
physical links conveying traffic in accordance with multiple transport protocols, 
the digital conveyance of the phase difference information may employ any 
physical link in the control services link 116. In accordance with an exemplary 
implementation, a control services link 116 includes, but is not limited to: an 
El/Tl trunk, an Ethernet link, an RS-485 differential serial link, etc. 

[42] In accordance with the exemplary embodiment of the invention, at each 
control card 112, digitally encoded phase difference information (phase error 
word streams) for each ESYNC signal selected at each peripheral shelf 106 is 
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provided to an ESYNC re-generator 126 associated with the control card 112. 
Each ESYNC re-generator 126 is directed by a selector 125 to select a particular 
stream of digital phase difference information (phase error word stream) 
associated with a corresponding selected ESYNC signal received at one of the 
peripheral shelves 106, and derive a corresponding ESYNC signal from the 
SSYNC signal. The derived ESYNC signals are provided to the SSU 122 as if 
received at the control shelf 102. 

[43] Each SSU 122 treats all ESYNC signals provided thereto equally and 
frequency locks and phase locks the SSYNC signal to a selected primary ESYNC 
signal as directed. 

[44] Therefore, in accordance with the exemplary embodiment of the 
invention, at the control shelf 102 any master clock signal from any interface 
card in the entire multi-shelf network node 100 can be selected as the primary 
and secondary master clock signals. The use of comparators 120 and ESYNC 
phase difference information encoders 124 at the peripheral shelf 106, and the 
use of ESYNC regenerators 126 at each control card 112 of the control shelf 102 
distributes clock signal processing in the multi-shelf network node system and 
results in a reduction in the number of clock signals to be provided to the 
control shelf 102 while still providing flexibility and redundancy. 

[45] In accordance with the exemplary embodiment of the invention, by 
detecting phase differences between ESYNC signals and the SSYNC signal at 
the peripheral shelves 106, instead of at the control shelf 102; jitter, skew, and 
noise distortions in providing ESYNC signals from the peripheral shelves 106 to 
the control shelf 102 are minimized. 

[46] In accordance with the exemplary embodiment of the invention, the 
digital transmission of phase difference information from each peripheral shelf 
106 to the control shelf 102, by exploiting existing control services links 116, 
eliminates the otherwise necessary additional wiring thereby reducing 
deployment and ongoing maintenance costs. 
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[47] In accordance with the exemplary embodiment of the invention, the re- 
generation of ESYNC signals at the control shelf 102, which are treated 
equivalent to ESYNC signals derived from master clock signals received 
directly at the control shelf 102, provides an SSU independent solution which 
allows a standard or commercial SSU 112 to be employed and/or upgraded 
independently. 

[48] The embodiments presented are exemplary only and persons skilled in 
the art would appreciate that variations to the above described embodiments 
may be made without departing from the spirit of the invention. The scope of 
the invention is solely defined by the appended claims. 
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